Using the Multidimensionality-Based DIF Analysis Paradigm to Study Cognitive Skills that Elicit Group Differences: A Critique

described a three-step approach used by researchers and practitioners attempting to identify biased test items: First, statistical methods are used to find items for which there are unexpected differences in performance between two groups (e.g., men and women). Second, each potentially biased item is examined for the reasons it is relatively more difficult for a particular group of examinees. Third, an item is considered to be biased if it can be established that the source of the unexpected or "extra" difficulty for one group is not relevant to what the test measures. In other words, this standard approach to DIF detection requires that each item is first tested statistically using a conditional DIF detection method (for a review of these methods, see Clauser & Mazor, 1998) and then scrutinized using some form of substantive review to identify the cause of the group difference. This approach has also been described as an exploratory DIF analysis, meaning items that produce unexpected group differences are flagged statistically and then scrutinized by reviewers who attempt to understand why the item may be more difficult for one group of examinees. Exploratory DIF analyses are often conducted when few a priori ideas exist about which items elicit group differences or why Despite its popularity and frequent use, researchers and practitioners also tend to agree that the standard approach to DIF detection has not increased our understanding about why group differences occur because statistically flagged DIF items are difficult to interpret. For example, Angoff (1993) claimed, more than a decade ago, that: "It has been reported by test developers that they are often confronted by DIF results that they cannot understand; and no amount of deliberation seems to help explain why some perfectly reasonable items have large DIF values" (p. 19). Camilli and Shepard (1994) reported that, in their experience, as many as half of the items with " large " DIF in any one study might not be interpretable. Roussos and Stout (1996), DIF Analysis Paradigm 3 after reviewing the DIF literature, concluded, " attempts at understanding the underlying causes of DIF using substantive analyses of statistically identified DIF items have, with few exceptions, met with overwhelming failure " (p. 360). The authors of the Standards for Educational and Psychological Testing (1999), which represent the most widely accepted standards for educational and psychological testing, summarized these concerns more succinctly when they stated: Although DIF procedures may …